{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Paddlepaddle实现浅层神经网络\n",
    "\n",
    "\n",
    "实践部分将搭建神经网络，包含一个隐藏层，实验将会展现出与Logistic回归的不同之处。\n",
    "\n",
    "实验将使用两层神经网络实现对“花”型图案的分类，如图所示，图中的点包含红点（y=0）和蓝点（y=1）还有点的坐标信息，实验将通过以下步骤完成对两种点的分类，使用PaddlePaddle实现。\n",
    "\n",
    "- 输入样本；\n",
    "\n",
    "- 搭建神经网络；\n",
    "\n",
    "- 初始化参数；\n",
    "\n",
    "- 训练，包括前向传播与后向传播（即BP算法）；\n",
    "\n",
    "- 得出训练后的参数；\n",
    "\n",
    "- 根据训练所得参数，绘制两类点边界曲线。\n",
    "\n",
    "<img src=\"image/data.png\" style=\"width:400px;height:300px;\">\n",
    "\n",
    "在该实验中，我们将使用PaddlePaddle实现浅层神经网络，解决识别猫的问题，使用的数据与“Paddlepaddle实现Logistic回归”中一致。\n",
    "\n",
    "本章节代码与“PaddlePaddle实现Logistic回归”基本一致，重复内容不再赘述，区别在于增加了一层隐藏层。\n",
    "\n",
    "## 1 - 引用库\n",
    "\n",
    "首先，载入几个需要用到的库，它们分别是：\n",
    "- numpy：一个python的基本库，用于科学计算\n",
    "- planar_utils：定义了一些工具函数\n",
    "- paddle.v2：paddle深度学习平台\n",
    "- matplotlib.pyplot：用于生成图，在验证模型准确率和展示成本变化趋势时会使用到"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import numpy as np\n",
    "import paddle.v2 as paddle\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import planar_utils\n",
    "\n",
    "TRAINING_SET = None\n",
    "TEST_SET = None\n",
    "DATADIM = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - 数据预处理\n",
    "\n",
    "定义load_data()用于载入数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 载入数据\n",
    "def load_data():\n",
    "    global TRAINING_SET, TEST_SET, DATADIM\n",
    "\n",
    "    train_set_x, train_set_y, test_set_x, test_set_y = planar_utils.load_planar_dataset()\n",
    "\n",
    "    # 定义纬度\n",
    "    DATADIM = 2\n",
    "\n",
    "    TRAINING_SET = np.hstack((train_set_x.T, train_set_y.T))\n",
    "    TEST_SET = np.hstack((test_set_x.T, test_set_y.T))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - 构造reader\n",
    "\n",
    "构造reader()函数来读取训练数据集TRAINING_SET和测试数据集TEST_SET，需要注意的是，yield关键字的作用类似return关键字，但不同指出在于yield关键字让reader()变成一个生成器（Generator），生成器不会创建完整的数据集列表，而是在每次循环时计算下一个值，这样不仅节省内存空间，而且符合reader的定义，也即一个真正的读取器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "# 读取训练数据或测试数据，服务于train()和test()\n",
    "def read_data(data_set):\n",
    "    \"\"\"\n",
    "        一个reader\n",
    "        Args:\n",
    "            data_set -- 要获取的数据集\n",
    "        Return:\n",
    "            reader -- 用于获取训练数据集及其标签的生成器generator\n",
    "    \"\"\"\n",
    "    def reader():\n",
    "        \"\"\"\n",
    "        一个reader\n",
    "        Args:\n",
    "        Return:\n",
    "            data[:-1], data[-1:] -- 使用yield返回生成器(generator)，\n",
    "                    data[:-1]表示前n-1个元素，也就是训练数据，data[-1:]表示最后一个元素，也就是对应的标签\n",
    "        \"\"\"\n",
    "        for data in data_set:\n",
    "            yield data[:-1], data[-1:]\n",
    "    return reader\n",
    "\n",
    "\n",
    "# 获取训练数据集\n",
    "def train():\n",
    "    \"\"\"\n",
    "    定义一个reader来获取训练数据集及其标签\n",
    "\n",
    "    Args:\n",
    "    Return:\n",
    "        read_data -- 用于获取训练数据集及其标签的reader\n",
    "    \"\"\"\n",
    "    global TRAINING_SET\n",
    "\n",
    "    return read_data(TRAINING_SET)\n",
    "\n",
    "\n",
    "# 获取测试数据集\n",
    "def test():\n",
    "    \"\"\"\n",
    "    定义一个reader来获取测试数据集及其标签\n",
    "\n",
    "    Args:\n",
    "    Return:\n",
    "        read_data -- 用于获取测试数据集及其标签的reader\n",
    "    \"\"\"\n",
    "    global TEST_SET\n",
    "\n",
    "    return read_data(TEST_SET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 - 配置网络结构和设置参数\n",
    "\n",
    "开始配置网络结构，这是本章与Logistic回归的不同之处，本章节实现双层神经网络，增加了一层隐藏层，隐藏层设置7个节点，激活函数使用Relu，其余不变。\n",
    "\n",
    "** 损失函数 **\n",
    "\n",
    "在这里使用PaddlePaddle提供的交叉熵损失函数，cost = paddle.layer.multi_binary_label_cross_entropy_cost(input=y_predict, label=y_label)定义了成本函数，并使用y_predict与label计算成本。定义了成本函数之后，使用PaddlePaddle提供的简单接口parameters=paddle.parameters.create(cost)来创建和初始化参数。\n",
    "\n",
    "** optimizer **\n",
    "\n",
    "参数创建完成后，定义参数优化器optimizer= paddle.optimizer.Momentum(momentum=0, learning_rate=0.00002)，使用Momentum作为优化器，并设置动量momentum为零，学习率为0.00002。注意，读者暂时无需了解Momentum的含义，只需要学会使用即可。\n",
    "\n",
    "** 其它配置 **\n",
    "\n",
    "feeding={‘image’:0, ‘label’:1}是数据层名称和数组索引的映射，用于在训练时输入数据。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 配置网络结构和设置参数\n",
    "def netconfig():\n",
    "    \"\"\"\n",
    "    配置网络结构和设置参数\n",
    "    Args:\n",
    "    Return:\n",
    "        image -- 输入层，DATADIM维稠密向量\n",
    "        y_predict -- 输出层，Sigmoid作为激活函数\n",
    "        y_label -- 标签数据，1维稠密向量\n",
    "        cost -- 损失函数\n",
    "        parameters -- 模型参数\n",
    "        optimizer -- 优化器\n",
    "        feeding -- 数据映射，python字典\n",
    "    \"\"\"\n",
    "    # 输入层，paddle.layer.data表示数据层,name=’image’：名称为image,\n",
    "    # type=paddle.data_type.dense_vector(DATADIM)：数据类型为DATADIM维稠密向量\n",
    "    image = paddle.layer.data(\n",
    "        name='image', type=paddle.data_type.dense_vector(DATADIM))\n",
    "\n",
    "    # 隐藏层，paddle.layer.fc表示全连接层，input=image: 该层输入数据为image\n",
    "    # size=4：神经元个数，act=paddle.activation.Tanh()：激活函数为Tanh()\n",
    "    h1 = paddle.layer.fc(\n",
    "        input=image, size=4, act=paddle.activation.Tanh())\n",
    "\n",
    "    # 输出层，paddle.layer.fc表示全连接层，input=h1: 该层输入数据为h1\n",
    "    # size=1：神经元个数，act=paddle.activation.Sigmoid()：激活函数为Sigmoid()\n",
    "    y_predict = paddle.layer.fc(\n",
    "        input=h1, size=1, act=paddle.activation.Sigmoid())\n",
    "\n",
    "    # 标签数据，paddle.layer.data表示数据层，name=’label’：名称为label\n",
    "    # type=paddle.data_type.dense_vector(1)：数据类型为1维稠密向量\n",
    "    y_label = paddle.layer.data(\n",
    "        name='label', type=paddle.data_type.dense_vector(1))\n",
    "\n",
    "    # 定义成本函数为交叉熵损失函数multi_binary_label_cross_entropy_cost\n",
    "    cost = paddle.layer.multi_binary_label_cross_entropy_cost(input=y_predict, label=y_label)\n",
    "\n",
    "    # 利用cost创建parameters\n",
    "    parameters = paddle.parameters.create(cost)\n",
    "\n",
    "    # 创建optimizer，并初始化momentum和learning_rate\n",
    "    optimizer = paddle.optimizer.Momentum(momentum=0, learning_rate=0.0075)\n",
    "\n",
    "    # 数据层和数组索引映射，用于trainer训练时喂数据\n",
    "    feeding = {\n",
    "        'image': 0,\n",
    "        'label': 1}\n",
    "\n",
    "    data = [image, y_predict, y_label, cost, parameters, optimizer, feeding]\n",
    "\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 - 训练过程\n",
    "\n",
    "接下来进入训练过程。\n",
    "\n",
    "** 初始化 **\n",
    "\n",
    "首先进行最基本的初始化操作，paddle.init(use_gpu=False, trainer_count=1)表示不使用gpu进行训练并且仅使用一个trainer进行训练，load_data()用于获取并预处理数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 初始化\n",
    "paddle.init(use_gpu=False, trainer_count=1)\n",
    "\n",
    "# 获取数据并预处理\n",
    "load_data()\n",
    "\n",
    "# 配置网络结构和设置参数\n",
    "image, y_predict, y_label, cost, parameters, optimizer, feeding = netconfig()\n",
    "\n",
    "# 记录成本cost\n",
    "costs = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** 模型训练 **\n",
    "\n",
    "上述内容进行了初始化并配置了网络结构，接下来利用上述配置进行模型训练。\n",
    "\n",
    "首先定义一个随机梯度下降trainer，配置三个参数cost、parameters、update_equation，它们分别表示成本函数、参数和更新公式。\n",
    "\n",
    "再利用trainer.train()即可开始真正的模型训练：\n",
    "- paddle.reader.shuffle(train(), buf_size=5000)表示trainer从train()这个reader中读取了buf_size=5000大小的数据并打乱顺序\n",
    "- paddle.batch(reader(), batch_size=256)表示从打乱的数据中再取出batch_size=256大小的数据进行一次迭代训练\n",
    "- 参数feeding用到了之前定义的feeding索引，将数据层image和label输入trainer，也就是训练数据的来源。\n",
    "- 参数event_handler是事件管理机制，读者可以自定义event_handler，根据事件信息作相应的操作。\n",
    "- 参数num_passes=5000表示迭代训练5000次后停止训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pass 0, Batch 0, Cost 0.772777\n",
      "Pass 0, Batch 1, Cost 0.684858\n",
      "Pass 100, Batch 0, Cost 0.299152\n",
      "Pass 100, Batch 1, Cost 0.335288\n",
      "Pass 200, Batch 0, Cost 0.320332\n",
      "Pass 200, Batch 1, Cost 0.187563\n",
      "Pass 300, Batch 0, Cost 0.306633\n",
      "Pass 300, Batch 1, Cost 0.209611\n",
      "Pass 400, Batch 0, Cost 0.288556\n",
      "Pass 400, Batch 1, Cost 0.317489\n",
      "Pass 500, Batch 0, Cost 0.287661\n",
      "Pass 500, Batch 1, Cost 0.250487\n",
      "Pass 600, Batch 0, Cost 0.273198\n",
      "Pass 600, Batch 1, Cost 0.285792\n",
      "Pass 700, Batch 0, Cost 0.265536\n",
      "Pass 700, Batch 1, Cost 0.271602\n",
      "Pass 800, Batch 0, Cost 0.263223\n",
      "Pass 800, Batch 1, Cost 0.204638\n",
      "Pass 900, Batch 0, Cost 0.240170\n",
      "Pass 900, Batch 1, Cost 0.290065\n",
      "Pass 1000, Batch 0, Cost 0.238204\n",
      "Pass 1000, Batch 1, Cost 0.257193\n",
      "Pass 1100, Batch 0, Cost 0.229301\n",
      "Pass 1100, Batch 1, Cost 0.277811\n",
      "Pass 1200, Batch 0, Cost 0.225052\n",
      "Pass 1200, Batch 1, Cost 0.300405\n",
      "Pass 1300, Batch 0, Cost 0.219078\n",
      "Pass 1300, Batch 1, Cost 0.294040\n",
      "Pass 1400, Batch 0, Cost 0.238975\n",
      "Pass 1400, Batch 1, Cost 0.216461\n",
      "Pass 1500, Batch 0, Cost 0.215900\n",
      "Pass 1500, Batch 1, Cost 0.298958\n",
      "Pass 1600, Batch 0, Cost 0.227579\n",
      "Pass 1600, Batch 1, Cost 0.248936\n",
      "Pass 1700, Batch 0, Cost 0.222204\n",
      "Pass 1700, Batch 1, Cost 0.257451\n",
      "Pass 1800, Batch 0, Cost 0.234150\n",
      "Pass 1800, Batch 1, Cost 0.197953\n",
      "Pass 1900, Batch 0, Cost 0.232878\n",
      "Pass 1900, Batch 1, Cost 0.223976\n",
      "Pass 2000, Batch 0, Cost 0.206835\n",
      "Pass 2000, Batch 1, Cost 0.316677\n",
      "Pass 2100, Batch 0, Cost 0.224995\n",
      "Pass 2100, Batch 1, Cost 0.224580\n",
      "Pass 2200, Batch 0, Cost 0.226882\n",
      "Pass 2200, Batch 1, Cost 0.255160\n",
      "Pass 2300, Batch 0, Cost 0.228322\n",
      "Pass 2300, Batch 1, Cost 0.202146\n",
      "Pass 2400, Batch 0, Cost 0.229634\n",
      "Pass 2400, Batch 1, Cost 0.269629\n",
      "Pass 2500, Batch 0, Cost 0.223437\n",
      "Pass 2500, Batch 1, Cost 0.270462\n",
      "Pass 2600, Batch 0, Cost 0.244703\n",
      "Pass 2600, Batch 1, Cost 0.242959\n",
      "Pass 2700, Batch 0, Cost 0.229759\n",
      "Pass 2700, Batch 1, Cost 0.187647\n",
      "Pass 2800, Batch 0, Cost 0.240064\n",
      "Pass 2800, Batch 1, Cost 0.195292\n",
      "Pass 2900, Batch 0, Cost 0.206180\n",
      "Pass 2900, Batch 1, Cost 0.274637\n",
      "Pass 3000, Batch 0, Cost 0.244419\n",
      "Pass 3000, Batch 1, Cost 0.183190\n",
      "Pass 3100, Batch 0, Cost 0.228286\n",
      "Pass 3100, Batch 1, Cost 0.215953\n",
      "Pass 3200, Batch 0, Cost 0.216068\n",
      "Pass 3200, Batch 1, Cost 0.230640\n",
      "Pass 3300, Batch 0, Cost 0.219326\n",
      "Pass 3300, Batch 1, Cost 0.223286\n",
      "Pass 3400, Batch 0, Cost 0.213912\n",
      "Pass 3400, Batch 1, Cost 0.253081\n",
      "Pass 3500, Batch 0, Cost 0.220520\n",
      "Pass 3500, Batch 1, Cost 0.223268\n",
      "Pass 3600, Batch 0, Cost 0.224340\n",
      "Pass 3600, Batch 1, Cost 0.208784\n",
      "Pass 3700, Batch 0, Cost 0.231136\n",
      "Pass 3700, Batch 1, Cost 0.224838\n",
      "Pass 3800, Batch 0, Cost 0.227527\n",
      "Pass 3800, Batch 1, Cost 0.242226\n",
      "Pass 3900, Batch 0, Cost 0.224787\n",
      "Pass 3900, Batch 1, Cost 0.185405\n",
      "Pass 4000, Batch 0, Cost 0.213723\n",
      "Pass 4000, Batch 1, Cost 0.299522\n",
      "Pass 4100, Batch 0, Cost 0.220463\n",
      "Pass 4100, Batch 1, Cost 0.214266\n",
      "Pass 4200, Batch 0, Cost 0.219302\n",
      "Pass 4200, Batch 1, Cost 0.217053\n",
      "Pass 4300, Batch 0, Cost 0.206765\n",
      "Pass 4300, Batch 1, Cost 0.251817\n",
      "Pass 4400, Batch 0, Cost 0.220867\n",
      "Pass 4400, Batch 1, Cost 0.193217\n",
      "Pass 4500, Batch 0, Cost 0.215219\n",
      "Pass 4500, Batch 1, Cost 0.247486\n",
      "Pass 4600, Batch 0, Cost 0.236024\n",
      "Pass 4600, Batch 1, Cost 0.146560\n",
      "Pass 4700, Batch 0, Cost 0.221007\n",
      "Pass 4700, Batch 1, Cost 0.229954\n",
      "Pass 4800, Batch 0, Cost 0.226011\n",
      "Pass 4800, Batch 1, Cost 0.169823\n",
      "Pass 4900, Batch 0, Cost 0.208402\n",
      "Pass 4900, Batch 1, Cost 0.298827\n",
      "Pass 5000, Batch 0, Cost 0.205629\n",
      "Pass 5000, Batch 1, Cost 0.246386\n",
      "Pass 5100, Batch 0, Cost 0.205989\n",
      "Pass 5100, Batch 1, Cost 0.245758\n",
      "Pass 5200, Batch 0, Cost 0.211753\n",
      "Pass 5200, Batch 1, Cost 0.292617\n",
      "Pass 5300, Batch 0, Cost 0.219334\n",
      "Pass 5300, Batch 1, Cost 0.189336\n",
      "Pass 5400, Batch 0, Cost 0.232362\n",
      "Pass 5400, Batch 1, Cost 0.211656\n",
      "Pass 5500, Batch 0, Cost 0.222069\n",
      "Pass 5500, Batch 1, Cost 0.189116\n",
      "Pass 5600, Batch 0, Cost 0.249768\n",
      "Pass 5600, Batch 1, Cost 0.222271\n",
      "Pass 5700, Batch 0, Cost 0.203444\n",
      "Pass 5700, Batch 1, Cost 0.257100\n",
      "Pass 5800, Batch 0, Cost 0.214192\n",
      "Pass 5800, Batch 1, Cost 0.205431\n",
      "Pass 5900, Batch 0, Cost 0.214307\n",
      "Pass 5900, Batch 1, Cost 0.219738\n",
      "Pass 6000, Batch 0, Cost 0.191089\n",
      "Pass 6000, Batch 1, Cost 0.427355\n",
      "Pass 6100, Batch 0, Cost 0.225279\n",
      "Pass 6100, Batch 1, Cost 0.271443\n",
      "Pass 6200, Batch 0, Cost 0.212175\n",
      "Pass 6200, Batch 1, Cost 0.223343\n",
      "Pass 6300, Batch 0, Cost 0.222561\n",
      "Pass 6300, Batch 1, Cost 0.170972\n",
      "Pass 6400, Batch 0, Cost 0.230605\n",
      "Pass 6400, Batch 1, Cost 0.177243\n",
      "Pass 6500, Batch 0, Cost 0.214216\n",
      "Pass 6500, Batch 1, Cost 0.235590\n",
      "Pass 6600, Batch 0, Cost 0.199382\n",
      "Pass 6600, Batch 1, Cost 0.277378\n",
      "Pass 6700, Batch 0, Cost 0.228598\n",
      "Pass 6700, Batch 1, Cost 0.155258\n",
      "Pass 6800, Batch 0, Cost 0.215724\n",
      "Pass 6800, Batch 1, Cost 0.197191\n",
      "Pass 6900, Batch 0, Cost 0.187020\n",
      "Pass 6900, Batch 1, Cost 0.309408\n",
      "Pass 7000, Batch 0, Cost 0.199948\n",
      "Pass 7000, Batch 1, Cost 0.263835\n",
      "Pass 7100, Batch 0, Cost 0.231008\n",
      "Pass 7100, Batch 1, Cost 0.241277\n",
      "Pass 7200, Batch 0, Cost 0.199381\n",
      "Pass 7200, Batch 1, Cost 0.321671\n",
      "Pass 7300, Batch 0, Cost 0.214993\n",
      "Pass 7300, Batch 1, Cost 0.200598\n",
      "Pass 7400, Batch 0, Cost 0.208654\n",
      "Pass 7400, Batch 1, Cost 0.217581\n",
      "Pass 7500, Batch 0, Cost 0.199258\n",
      "Pass 7500, Batch 1, Cost 0.261730\n",
      "Pass 7600, Batch 0, Cost 0.193863\n",
      "Pass 7600, Batch 1, Cost 0.318485\n",
      "Pass 7700, Batch 0, Cost 0.218621\n",
      "Pass 7700, Batch 1, Cost 0.214984\n",
      "Pass 7800, Batch 0, Cost 0.218014\n",
      "Pass 7800, Batch 1, Cost 0.225268\n",
      "Pass 7900, Batch 0, Cost 0.214022\n",
      "Pass 7900, Batch 1, Cost 0.216392\n",
      "Pass 8000, Batch 0, Cost 0.202706\n",
      "Pass 8000, Batch 1, Cost 0.247205\n",
      "Pass 8100, Batch 0, Cost 0.206601\n",
      "Pass 8100, Batch 1, Cost 0.274779\n",
      "Pass 8200, Batch 0, Cost 0.222018\n",
      "Pass 8200, Batch 1, Cost 0.165801\n",
      "Pass 8300, Batch 0, Cost 0.211870\n",
      "Pass 8300, Batch 1, Cost 0.205229\n",
      "Pass 8400, Batch 0, Cost 0.197649\n",
      "Pass 8400, Batch 1, Cost 0.258502\n",
      "Pass 8500, Batch 0, Cost 0.215542\n",
      "Pass 8500, Batch 1, Cost 0.212061\n",
      "Pass 8600, Batch 0, Cost 0.212647\n",
      "Pass 8600, Batch 1, Cost 0.201391\n",
      "Pass 8700, Batch 0, Cost 0.218901\n",
      "Pass 8700, Batch 1, Cost 0.176204\n",
      "Pass 8800, Batch 0, Cost 0.225683\n",
      "Pass 8800, Batch 1, Cost 0.189363\n",
      "Pass 8900, Batch 0, Cost 0.250457\n",
      "Pass 8900, Batch 1, Cost 0.195355\n",
      "Pass 9000, Batch 0, Cost 0.209093\n",
      "Pass 9000, Batch 1, Cost 0.223908\n",
      "Pass 9100, Batch 0, Cost 0.202863\n",
      "Pass 9100, Batch 1, Cost 0.253735\n",
      "Pass 9200, Batch 0, Cost 0.208534\n",
      "Pass 9200, Batch 1, Cost 0.209088\n",
      "Pass 9300, Batch 0, Cost 0.217119\n",
      "Pass 9300, Batch 1, Cost 0.189176\n",
      "Pass 9400, Batch 0, Cost 0.195976\n",
      "Pass 9400, Batch 1, Cost 0.272448\n",
      "Pass 9500, Batch 0, Cost 0.222696\n",
      "Pass 9500, Batch 1, Cost 0.190952\n",
      "Pass 9600, Batch 0, Cost 0.224336\n",
      "Pass 9600, Batch 1, Cost 0.257351\n",
      "Pass 9700, Batch 0, Cost 0.218235\n",
      "Pass 9700, Batch 1, Cost 0.176171\n",
      "Pass 9800, Batch 0, Cost 0.208494\n",
      "Pass 9800, Batch 1, Cost 0.234269\n",
      "Pass 9900, Batch 0, Cost 0.212364\n",
      "Pass 9900, Batch 1, Cost 0.202101\n"
     ]
    }
   ],
   "source": [
    "def event_handler(event):\n",
    "    \"\"\"\n",
    "    事件处理器，可以根据训练过程的信息作相应操作\n",
    "\n",
    "    Args:\n",
    "        event -- 事件对象，包含event.pass_id, event.batch_id, event.cost等信息\n",
    "    Return:\n",
    "    \"\"\"\n",
    "    if isinstance(event, paddle.event.EndIteration):\n",
    "        if event.pass_id % 100 == 0:\n",
    "            print(\"Pass %d, Batch %d, Cost %f\" % (event.pass_id, event.batch_id, event.cost))\n",
    "            costs.append(event.cost)\n",
    "#             with open('params_pass_%d.tar' % event.pass_id, 'w') as f:\n",
    "#                 parameters.to_tar(f)\n",
    "\n",
    "# 构造trainer,配置三个参数cost、parameters、update_equation，它们分别表示成本函数、参数和更新公式。\n",
    "trainer = paddle.trainer.SGD(\n",
    "    cost=cost, parameters=parameters, update_equation=optimizer)\n",
    "\n",
    "\"\"\"\n",
    "模型训练\n",
    "paddle.reader.shuffle(train(), buf_size=5000)：表示trainer从train()这个reader中读取了buf_size=5000\n",
    "大小的数据并打乱顺序\n",
    "paddle.batch(reader(), batch_size=256)：表示从打乱的数据中再取出batch_size=256大小的数据进行一次迭代训练\n",
    "feeding：用到了之前定义的feeding索引，将数据层image和label输入trainer\n",
    "event_handler：事件管理机制，可以自定义event_handler，根据事件信息作相应的操作\n",
    "num_passes：定义训练的迭代次数\n",
    "\"\"\"\n",
    "trainer.train(\n",
    "    reader=paddle.batch(\n",
    "        paddle.reader.shuffle(train(), buf_size=5000),\n",
    "        batch_size=256),\n",
    "    feeding=feeding,\n",
    "    event_handler=event_handler,\n",
    "    num_passes=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** 模型检验 **\n",
    "\n",
    "模型训练完成后，接下来检验模型的准确率。\n",
    "\n",
    "首先定义get_data()函数来帮助我们读取训练数据和测试数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 获取data\n",
    "def get_data(data_creator):\n",
    "    \"\"\"\n",
    "    使用参数data_creator来获取测试数据\n",
    "\n",
    "    Args:\n",
    "        data_creator -- 数据来源,可以是train()或者test()\n",
    "    Return:\n",
    "        result -- 包含测试数据(image)和标签(label)的python字典\n",
    "    \"\"\"\n",
    "    data_creator = data_creator\n",
    "    data_image = []\n",
    "    data_label = []\n",
    "\n",
    "    for item in data_creator():\n",
    "        data_image.append((item[0],))\n",
    "        data_label.append(item[1])\n",
    "\n",
    "    result = {\n",
    "        \"image\": data_image,\n",
    "        \"label\": data_label\n",
    "    }\n",
    "\n",
    "    return result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获得数据之后，我们就可以开始利用paddle.infer()来进行预测，参数output_layer 表示输出层，参数parameters表示模型参数，参数input表示输入的测试数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 获取测试数据和训练数据，用来验证模型准确度\n",
    "train_data = get_data(train())\n",
    "test_data = get_data(test())\n",
    "    \n",
    "# 根据train_data和test_data预测结果，output_layer表示输出层，parameters表示模型参数，input表示输入的测试数据\n",
    "probs_train = paddle.infer(\n",
    "    output_layer=y_predict, parameters=parameters, input=train_data['image']\n",
    ")\n",
    "probs_test = paddle.infer(\n",
    "    output_layer=y_predict, parameters=parameters, input=test_data['image']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获得检测结果probs_train和probs_test之后，我们将结果转化为二分类结果并计算预测正确的结果数量，定义calc_accuracy()分别计算训练准确度和测试准确度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 计算准确度\n",
    "def calc_accuracy(probs, data):\n",
    "    \"\"\"\n",
    "    根据数据集来计算准确度accuracy\n",
    "\n",
    "    Args:\n",
    "        probs -- 数据集的预测结果，调用paddle.infer()来获取\n",
    "        data -- 数据集\n",
    "\n",
    "    Return:\n",
    "        calc_accuracy -- 训练准确度\n",
    "    \"\"\"\n",
    "    right = 0\n",
    "    total = len(data['label'])\n",
    "    for i in range(len(probs)):\n",
    "        if float(probs[i][0]) > 0.5 and data['label'][i] == 1:\n",
    "            right += 1\n",
    "        elif float(probs[i][0]) < 0.5 and data['label'][i] == 0:\n",
    "            right += 1\n",
    "    accuracy = (float(right) / float(total)) * 100\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "调用上述两个函数并输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_accuracy: 90.625 %\n",
      "test_accuracy: 90.75 %\n"
     ]
    }
   ],
   "source": [
    "# 计算train_accuracy和test_accuracy\n",
    "print(\"train_accuracy: {} %\".format(calc_accuracy(probs_train, train_data)))\n",
    "print(\"test_accuracy: {} %\".format(calc_accuracy(probs_test, test_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** 学习曲线 **\n",
    "\n",
    "可以输出成本的变化情况，利用学习曲线对模型进行分析。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzsnXecXFd597/P9Nle1VararnIBRfZuGGMAWNTbHrs4AAB4hBikjeQEBMSYsxLAiGExMEJvYUXjDEYbCJjio17kVxlVatrV1ppe50+5/3jlr0zO7N9dlea5/v5zGd37tyZe+bOved3nnKeI8YYFEVRFAXAN98NUBRFURYOKgqKoiiKi4qCoiiK4qKioCiKorioKCiKoiguKgqKoiiKi4qCclwjIveJyPvmux2KcqKgoqBMCxHZLyKvm+92GGOuNsZ8b77bASAivxeRD83DcRtE5G4RGRaRAyLyh+PsKyLyBRHpth9fEBHxvH62iDwjIiP237M9r90nIkOeR1JEtnhe3y8iMc/rvy7dt1ZKhYqCsmARkcB8t8FhIbWlALcDSWAx8B7gv0Xk9CL73gi8FXgFcBbwFuBPAUQkBPwC+AFQD3wP+IW93RHgKucBPA78JO/z3+LZ58rZ/JLK3KCioMw6IvJmEXleRPpE5HEROcvz2s0iskdEBkVkm4i8zfPa+0XkMRH5soh0A7fY2x4VkX8VkV4R2SciV3ve447OJ7HvahF52D72b0XkdhH5QZHvcLmItInI34pIB/AdEakXkV+KSKf9+b8UkeX2/p8DXgV8xR4lf8XefqqI/EZEekRkp4i8e5bPdSXwDuAfjDFDxphHgXuAPyrylvcBXzLGtBlj2oEvAe+3X7scCAD/boxJGGNuAwS4osBxV2F93+/P2pdRFgQqCsqsIiLnAN/GGn02Al8D7hGRsL3LHqzOpBb4DPADEVnq+YhXAnuxRr2f82zbCTQB/wJ8y+vyyGO8fX8IPG236xaKd5wOS4AGYCXWCNsHfMd+vgKIAV8BMMZ8CngEuMkeJd9kd9i/sY+7CLgO+C8RWV/oYCLyX7aQFnq8WKSNJwNpY8wuz7YXgGKWwun264X2PR140eTWvnmxyGe9F3jEGLM/b/v/s0Xz1yLyiiJtUBYwKgrKbHMj8DVjzFPGmIzt708AFwIYY35ijDlsjMkaY34MvAxc4Hn/YWPMfxpj0saYmL3tgDHmG8aYDJZLYymWaBSi4L4isgI4H/i0MSbpGVGPRxb4R3vUHDPGdBtjfmqMGTHGDGKJ1qvHef+bgf3GmO/Y3+c54KfAuwrtbIz5iDGmrsjjrELvAaqAgbxt/UD1OPv35+1bZQtn/mvjfdZ7ge/mbXsPsApLNB8E7heRuiLtUBYoKgrKbLMS+Lh3lAu0AssAROS9HtdSH3AG1qje4VCBz+xw/jHGjNj/VhU5frF9lwE9nm3FjuWl0xgTd56ISIWIfM0O5g4ADwN1IuIv8v6VwCvzzsV7sCyQ2WIIqMnbVgMMTnL/GmDItg4m9VkicinWd7jLu90Y85gtniPGmH8G+rCsQuU4QkVBmW0OAZ/LG+VWGGN+JCIrgW8ANwGNxpg64CUsv7VDqcr2HgEaRKTCs611gvfkt+XjwCnAK40xNcBl9nYpsv8h4KG8c1FljPmzQgcTka/mZfd4H1uLtHEXEBCRdZ5trwCK7b/Vfr3QvluBs/Jcc2cV+Kz3AT8zxgwVOYaDIfe3VY4DVBSUmRAUkYjnEcDq9D8sIq8Ui0oReZOIVAOVWB1FJ4CI/DGWpVByjDEHgM1YweuQiFyElXkzFaqx4gh9ItIA/GPe60eBNZ7nvwROFpE/EpGg/ThfRE4r0sYPe7N78h4FYwTGmGHgZ8Ct9rm+BLgW+J8i3+H7wMdEpEVElmEJ3Xft134PZIC/EJGwiNxkb3/AebOIRIF3k+c6EpEVInKJfW4jIvI3WBbgY0XaoSxQVBSUmbARq5N0HrcYYzYDf4IVgO0FdmNntxhjtmFluzyB1YGeydx2Gu8BLgK6gf8L/Bgr3jFZ/h2IAl3Ak8Cv8l7/D+CddmbSbXbc4UqsAPNhLNfWF4Aws8tH7HYdA34E/JkxZiuAiLxKRLwj+q8B9wJbsKy0/7W3YYxJYqWrvhfL9fMB4K32doe32q89mNeGauC/sX7zduAq4GpjTPfsfU1lLhBdZEcpV0Tkx8AOY0z+iF9Ryha1FJSywXbdrBURn4hcheVm+fl8t0tRFhILeZamosw2S7D8741AG5ab5bn5bZKiLCxK6j6yR2P/AfiBbxpjPp/3+gqsXPI6e5+bjTEbS9YgRVEUZVxKJgp27vYu4PVYo7JNwPV2sNHZ5+vAc8aY/7ZneW40xqwqSYMURVGUCSml++gCYLcxZi+AiNyB5cPd5tnHMDpZphYrQ2NcmpqazKpVq2a3pYqiKCc4zzzzTJcxpnmi/UopCi3kzhhtw6pL4+UW4Nci8lGsHPYJSzGvWrWKzZs3z1YbFUVRygIROTCZ/eY7++h64LvGmOXAG4H/EZExbRKRG0Vks4hs7uzsnPNGKoqilAulFIV2cssILLe3efkgcCeAMeYJIEJuHRzs175ujNlgjNnQ3Dyh9aMoiqJMk1KKwiZgnVg17ENYszrzq1IeBF4LYE/9j2CXQFAURVHmnpKJgjEmjVX47H5gO3CnMWariNwqItfYu30c+BMReQFrev77jU6xVhRFmTdKOnnNnnOwMW/bpz3/bwMuKWUbFEVRlMkz34FmRVEUZQGhoqAoiqK4lI0obNrfwxfv30E2qyELRVGUYpSNKLxwqI/bH9zDcDI9301RFEVZsJSNKFSFrZj6UEJFQVEUpRjlIwoRSxQG4yoKiqIoxSgfUQirKCiKokxE2YhCdSQIqPtIURRlPMpIFOyYgloKiqIoRSkbURgNNKfmuSWKoigLl/IRBQ00K4qiTEjZiEJlSFNSFUVRJqJsRMHvEypDfrUUFEVRxqFsRAEsF5IGmhVFUYpTXqIQDqj7SFEUZRzKSxQiQQZVFBRFUYpSVqJQEwkwFNeUVEVRlGKUlSio+0hRFGV8yk8UNNCsKIpSlPIShUhAYwqKoijjUFaiUG27j3T1NUVRlMKUlShURQIYAyOpzHw3RVEUZUFSXqIQtstna1xBURSlICUVBRG5SkR2ishuEbm5wOtfFpHn7ccuEekrZXuconhaKVVRFKUwgVJ9sIj4gduB1wNtwCYRuccYs83ZxxjzV579PwqcU6r2gBVTAK2UqiiKUoxSWgoXALuNMXuNMUngDuDacfa/HvhRCdszutCOZiApiqIUpJSi0AIc8jxvs7eNQURWAquBB4q8fqOIbBaRzZ2dndNuUJWuvqYoijIuCyXQfB1wlzGmYFqQMebrxpgNxpgNzc3N0z6Is/qazlVQFEUpTClFoR1o9Txfbm8rxHWU2HUEUG1nH2lMQVEUpTClFIVNwDoRWS0iIayO/578nUTkVKAeeKKEbQGgMuwH1H2kKIpSjJKJgjEmDdwE3A9sB+40xmwVkVtF5BrPrtcBdxhjSj7NOOD3EQ36GdRKqYqiKAUpWUoqgDFmI7Axb9un857fUso25BMJ+kiks3N5SEVRlOOGhRJonjP8PiFTeqNEURTluKTsRMEnogXxFEVRilB2ouD3CRkVBUVRlIKUnSj4RN1HiqIoxSg7UQj41VJQFEUpRtmJgl9UFBRFUYpRdqLg8wlZdR8piqIUpOxEQS0FRVGU4pSfKPiEjM5dUxRFKUiZioKqgqIoSiHKThR8PiGj3iNFUZSClJ0o+AWd0awoilKE8hMFndGsKIpSFBUFRVEUxaU8RUHnKSiKohSk7ETBp/MUFEVRilJ2ouDXGc2KoihFKT9RUEtBURSlKOUnChpoVhRFKYqKgqIoiuJSdqLg0+wjRVGUopSdKPh1jWZFUZSilJ0oBNRSUBRFKUpJRUFErhKRnSKyW0RuLrLPu0Vkm4hsFZEflrI9YLuPtCKeoihKQQKl+mAR8QO3A68H2oBNInKPMWabZ591wCeBS4wxvSKyqFTtcfCLWgqKoijFKKWlcAGw2xiz1xiTBO4Ars3b50+A240xvQDGmGMlbA9gWwq6nIKiKEpBSikKLcAhz/M2e5uXk4GTReQxEXlSRK4q9EEicqOIbBaRzZ2dnTNqlN+HzmhWFEUpwnwHmgPAOuBy4HrgGyJSl7+TMebrxpgNxpgNzc3NMzugz6fzFBRFUYpQSlFoB1o9z5fb27y0AfcYY1LGmH3ALiyRKBlaEE9RFKU4pRSFTcA6EVktIiHgOuCevH1+jmUlICJNWO6kvSVsE34fKgqKoihFKJkoGGPSwE3A/cB24E5jzFYRuVVErrF3ux/oFpFtwIPA3xhjukvVJtAZzYqiKONRspRUAGPMRmBj3rZPe/43wMfsx5ygM5oVRVGKM9+B5jkn4BPSKgqKoigFKTtR8PkEQK0FRVGUApSdKPjFEgWNKyiKooyl7ETBsRQ0A0lRFGUsZScKfsd9pJaCoijKGMpOFAK2KGiwWVEUZSxlJwo+0UCzoihKMcpOFPwaU1AURSlK2YmCG2jWmIKiKMoYyk4UAu48hXluiKIoygKk7ETBmaeQVlVQFEUZQ9mJgk8tBUVRlKKUnSj47W+sMQVFUZSxlJ0oOCmpmn2kKIoylrIThYDP+so6o1lRFGUsZScKjvsonVFRUBRFyafsRMGd0ayWgqIoyhjKThR0RrOiKEpxyk4UdEazoihKccpOFAJqKSiKohSl7ETBrympiqIoRSk7UdA1mhVFUYpTUlEQkatEZKeI7BaRmwu8/n4R6RSR5+3Hh0rZHvAEmjWmoCiKMoZAqT5YRPzA7cDrgTZgk4jcY4zZlrfrj40xN5WqHflo9pGiKEpxSmkpXADsNsbsNcYkgTuAa0t4vEmhMQVFUZTilFIUWoBDnudt9rZ83iEiL4rIXSLSWuiDRORGEdksIps7Oztn1Ci1FBRFUYoz34Hme4FVxpizgN8A3yu0kzHm68aYDcaYDc3NzTM6oM5oVhRFKU4pRaEd8I78l9vbXIwx3caYhP30m8B5JWwP4LUUSn0kRVGU449SisImYJ2IrBaREHAdcI93BxFZ6nl6DbC9hO0BNPtIURRlPEqWfWSMSYvITcD9gB/4tjFmq4jcCmw2xtwD/IWIXAOkgR7g/aVqj8OopaCmgqIoSj6TEgUReZcx5icTbcvHGLMR2Ji37dOe/z8JfHLyzZ05o9lHc3lURVGU44PJuo8Kddxz2pnPFvYaOzqjWVEUpQDjWgoicjXwRqBFRG7zvFSD5fI57tCYgqIoSnEmch8dBjZjBYGf8WwfBP6qVI0qJTpPQVEUpTjjioIx5gXgBRH5oTEmBSAi9UCrMaZ3Lho42+iMZkVRlOJMNqbwGxGpEZEG4FngGyLy5RK2q2SopaAoilKcyYpCrTFmAHg78H1jzCuB15auWaXDLZ2tMQVFUZQxTFYUAvZEs3cDvyxhe0qOuo8URVGKM1lRuBVrEtoeY8wmEVkDvFy6ZpUOx32UVlFQFEUZw6Qmr9mT1H7ieb4XeEepGlVK/LrymqIoSlEmZSmIyHIRuVtEjtmPn4rI8lI3rhS47iONKSiKooxhsu6j72AVs1tmP+61tx136BrNiqIoxZmsKDQbY75jjEnbj+8CM1vYYB4J+EQtBUVRlAJMVhS6ReQGEfHbjxuA7lI2rJT4fKKBZkVRlAJMVhQ+gJWO2gEcAd7JHJS5LhV+EXUfKYqiFGCy6yncCrzPKW1hz2z+VyyxOO7w+0RLZyuKohRgspbCWd5aR8aYHuCc0jSp9PhEZzQriqIUYrKi4LML4QGupVCyVdtKTcDv0xnNygnHcCLN/Vs75rsZynHOZEXhS8ATIvJZEfks8DjwL6VrVmnxiQaalROP/91yhD/9n2c4NhCf76YoxzGTndH8fRHZDFxhb3q7MWZb6ZpVWvw+naegnHjEUxkAYvZfRZkOk3YB2SJw3AqBF7/oPAXlxCOZtrInUppFocyAybqPTih8Pk1JVU48HJdoMq3XtjJ9ylIUAjp5TTkBSdsWQlItBWUGlKUo+LTMhXICksoY+6+KgjJ9SioKInKViOwUkd0icvM4+71DRIyIbChlexx0RrNyIuKIQSqtoqBMn5KJgoj4gduBq4H1wPUisr7AftXAXwJPlaot+VgzmlUUlBMLN6agloIyA0ppKVwA7DbG7DXGJIE7gGsL7PdZ4AvAnCVX+0R0RrNywuFaChm9tpXpU0pRaAEOeZ632dtcRORcoNUY87/jfZCI3Cgim0Vkc2dn54wbFvBroFk58UhrTEGZBeYt0CwiPuDfgI9PtK8x5uvGmA3GmA3NzTNfxsEn6j5STjxGLQUVBWX6lFIU2oFWz/Pl9jaHauAM4Pcish+4ELhnLoLNfp+6j5QTD8dtlNBAszIDSikKm4B1IrJaRELAdVhLegJgjOk3xjQZY1YZY1YBTwLXGGM2l7BNgD2jWS0F5QQjnVVLQZk5JRMFY0wauAm4H9gO3GmM2Soit4rINaU67mTw+4Ss3jfKCYYbU1BLQZkBJS1/bYzZCGzM2/bpIvteXsq2ePH7hERai4YpJxZJzT5SZoEyntE8361QlNlFy1wos0FZioJftHS2cuLhpFlrTEGZCeUpCjqjWTkB0ZRUZTYoW1HQlFTlRMOJJSQ10KzMgLIVBZ3RrJxopDXQrMwCZSkKPq2SqpyAuJaCuo+UGVCWouDX9RSUExB38pq6j5QZUJ6ioDOalRMQXWRHmQ3KUxQ0+0g5AdHS2cpsoKKgKCcIaY0pKLNAWYqCT1NSlRMQLYinzAZlKQoaU1BORJz5CTpPQZkJ5SkK6j5STkC0zIUyG5S1KLzU3s/hvth8N0dRZoXRmIIOeJTpU76iYAw3fn8zN/9sy3w3R1FmjDGGlM5TUGaBkq6nsFDxiZBMZzncH+fYYIL+kRS1FcH5bpaiTJtM1uDkTqj7SJkJZWopgBNSSGcNv91+dH4bpCgzxFvLS0VBmQllKgo+z//CfS91zGNrFGXmeIVgIU1ey2aN1hk7zihPURBx/3/zWUt5+OVOhhLpeWyRosyMtEcIEgsopvCZe7fywe9tmvT+xqiIzDflKQr2t64KB7hy/RKS6SyHekYK7nvn5kN89aE9c9i62eeZA708c6B3vpuhlBDHUgj4ZEG5j/Z2DbOva3jS+//yxSOc/7nf6hrq80hZioLPZ1kKrQ0VREPWKYinCl+EP3jyAN96dN+cta0UfOFXO/jCfTvmuxlKCUnZo+uKkH9BiUI8lWE4OfkOfn/XMN3DSfpjqRK2ShmPshQFx320sqGCSNAPQDw19kbKZg0vHx2i085QOl4ZiqcZiB+/7VcmxllgpyIUmDNR6BlOMpIc3+0aS2WITUEUYvbgbCShlsJ8UVJREJGrRGSniOwWkZsLvP5hEdkiIs+LyKMisr6U7XHw25bCikaPKKQzGGN4el+Pu197X8y9SHd3Ds5F00pCLJXRmMkJjhNcrgj7SWUMZg5qe73v209PaIHGkhmGk+lJt8e53/R6nT9KJgoi4gduB64G1gPXF+j0f2iMOdMYczbwL8C/lao9Xvwe91EkYItCMsMTe7t599ee4KX2fgB2HR0VgpePDs1F00rCSDKtN9kJTsq1FPz289KLQsdAnI6B+Lj7xFNZjClsiRfe37YUpmBdKLNLKS2FC4Ddxpi9xpgkcAdwrXcHY8yA52klMCdpB44orGyoIBoatRR6hy0XS/dwEoBdthCE/D52HyuNKBhjeGhXZ0kzLkYSGYYTkx+tnQhks8Z1qZQDTvZRRdCajzoXLqR4MjNh5+26gyZwM7n72583PMn9ldmnlKLQAhzyPG+zt+UgIn8uInuwLIW/KPRBInKjiGwWkc2dnZ0zbljITj9a2VhBJOgEmrMef6Z1Qb58dJAlNRHWLa7i5RKJwott/bzv20/zxN7uovukM1ne/bUneOTlqX93YwwjqQypjFlQqYql5qsP7+FNtz06J8fKZA3v/87TPL6na06OVwinxIUzyJmLSqmx1MSi4IjBZEf+zj04rJbtvDHvgWZjzO3GmLXA3wJ/X2SfrxtjNhhjNjQ3N8/4mFefuZT/vP4cVjZWuu6jWDIzxp+569gg6xZXcdKiqpJZCl1DiZy/j+/uGjPC7R1J8fS+Hjbtn3paaTKTdSvCltONtufYMLs7h+bEOuqPpfj9zs6ceNRc41gKlWHHfVRaUUhlsqSzZtzOPps1rtto8qJg76+B5nmjlKLQDrR6ni+3txXjDuCtJWyPS200yFtesQwgx30Us0c1w4k02axh97EhTl5czbpFVbT3xabcqRpjiqa6Ojipd4PxNHs7h/jDbz7Fr7fllt0YtDOHBqaRpue9ucoprjCUSJGZoNOatWPFR6+b8fibn7zAdx8rTXpzypN9BKVffW0ybiGvZTpZd1A8qYHm+aaUorAJWCciq0UkBFwH3OPdQUTWeZ6+CXi5hO0pSDjgcR8lrYt4OJnhUO8I8VSWk21LAWBP59Sshd9sO8p5n/3NuOmsTkc/EE/RNWTFMvLLeQ/anc50crdHUuUpCs45c/6WEue8TnR+H9x5jMf3FHcTzoS5DjQ7vv/xRDfmufYmO/KfagxCmX1KJgrGmDRwE3A/sB240xizVURuFZFr7N1uEpGtIvI88DHgfaVqTzFEhEjQRzyVYSQ1enMf7rOyKlrrKwqKwv1bO/jsL7eN+9nbjwwynMzQ1ld4tjRAf2y083I6fSfQ7eB0bH0judsnQ8xzcw3NQQe5UHA66LmYnzEqCuN3fAOe33i2cdxH0dDcuI8cURhvDkKOKEw20Oy6cNV9NF+UtHS2MWYjsDFv26c9//9lKY8/WSJBP/FUBqci0nAi7bpsaqJBltdXAHCoxxrBG2P44v072X1siD9/zUk0VIYKfu6xQUtYOgcTRY/tdFqD8ZRrNXTl7e+0ZTodynCZuo8cIZ2Oy22qDCWsY4znPoqnMiTT2dKJgh1ornTcRyUONLsBYXsOgnjqibn7JL2iMElLIamWwnwz74HmhUAkYInCaOZDhgG7U6mNBokE/SyuCbv1kZ492OcGnjftLx5cdMSgkCjsta0Ob0yhqKVgdzZ903EfJctbFObCfeQcY7zzW2qRclZbq5glSyFhT+YshnOvGFO8AN90RCHuuQeV+UFFAcvkjqWy7oU7nEi7N29NxFp8p7W+goO2KPxk8yGiQT+hgG/cjJNjjigM5YrCU3u7ueJLD7H9yMBoTCGWckWhayjfUph+hxJLedxHcyQKe6cYeykFbnB+DtxHTgc2nntuJtbeZHAy1qKzEFNIprNc9M8P8LNni+eFxCfR4c/EfTTdTLntRwa0yuoMUVHACjbHU5nRUUpytFZQVcQyx1c0VNDWGyOWzHDvC4d581lLObu1jk37e4glMwUrQRazFJx9D3SP5FgKzjG7h/JjCqMdylRTLL0jrrlISd3ZMcgVX3poXnP2k+msO3qdU/fROB2fY3kOJzMl8fen8yyFZDrLge7JVyf10hdL0jOcZPuRgaL7eDv8YtdV7j4Tj/yNMTluqalyqGeEq//jkTHZewuFnuEk331s34KfRKqiwGhMYcSTDjcYT1MVDrizn5c3VHC4P8azB3sZTma46owlXLCqga2HB7jhW09x5ZcfyglEG2OKioLXghiIjw00dw4lci4cx1JIZaaeYuk14ScbaH6pvZ/bH9wNwEO7Onn7fz02aR91ux1Ud0qFzAdei2hgLrKPJpGSOuixWEohVMm8lNRfb+vg8n/9fdGS8OMxYCc/HBsnFubt8GNF0q5z3EepiX+HRDrrLik6nQGMU3KjrXfq33ku+N8tR7jl3m3s716Y7XNQUQCiwfyYguU+qomMxuFb66MYY/2wAGe31nHB6gYyWcMzB3oRkZxspL6RlHujjhUF6+LtGkzkpKQ6N2Mync3p2LwdylTdD16zfbIZHT/ZfIgv3r+TWDLDE3u6efZgHy8fm1xBwD47/bZUk/0mg1f85ib7aOLcem9soxQupHReSuqOI4MYA229sfHeVhDnnB0dp67RyCTcR/EppqQmPPWRpjO/xLn2uoamnqU3Fzj3+njndSGgogB2Smp2tO5KIsNAPEW1HU8Ay30EcN+WI7Q2RGmsCnPuynqqIwFuvGwNn3jDKfx+Zye/33kMGB1lBXwyJqZwdGB0FvOA133k6Sy8F7a3Q+mbYglvp5Z9XUXQdXNMRLs9T6JjIM4x+wLedri4K8HLQhCFwYR3VD4X8xSs48VT2aL1lmYi7JMh7a6nYA1kDtmj5Z7hqXeQznU4XtZcboef5su/2cX3Ht+fs48zyAr4ZFx3UDpjZWXluJum4T5yUrbzY3ILBee8jmeBLQRUFBh1H3n9mYPxNDVRj6Vgi0LvSIqzW+sBa+W2TZ96HX/3xtN470WriAR9PPqy5Ut3bqiTFlWNSTF1LoqjA3EGE2n8PmEokaZ3JEnULuXd7bmwZzLKjCUziEBjZWjSGR3O6PKopwrmtnH8y16cDKndx+amxEQhBufcUhg9XrFz7BWnUohCKi+m4PxuPcNT74Acl9t4I9r8zKJfPN/OT59ty9nHGe03VIbGnc/w/ScOcMW//t4VgupwYFrZR8USNfIZSqTnpViicy0eU0th4RMN+nMWAxlOWEHfGo+lsLgm4hbSe8XyWne7sx5DKOCjsTLsjswcF9Hpy2oZiKfZfWyIv/rx88SSGfei2GsHnJfURABrJvOa5kog98IejKdorg4DYzuUT929hftsl1YhRpIZKkMBqiJBN7XVIZs1BYOejqVwdCDudgxbJ20pWN9/IJ4eYyHNFY77KOCTOU1JBRgqMsKdqaXw5N5uvvJA8Qn/zu/ozFNw9Dg/vXkyOCPa4WTxdThyMotSGXpHUuzrHM4ZCDjWRGNVeNzV1/bZq605A6nGqtC0Ygqj7qPi150xhiv+9fd857H9U/78meIMDAq5kz/0vc30TuO3KgUqCkA46M9xH6Uyhu6hJDXRUVHw+4SW+igA56yoK/g5jVUh9yZ0rIHTl9UA8LWH9nD3c+1sPtDjXhQH7YDTcvtzh5MZ1jRbs6fz3Uet9j79sdHtiXRPz3LJAAAgAElEQVSGHz59kB9v9hajzWUkmSYa8lMV9o+50f5p43be+dUncrYNxFNuJ9fRH+eY7erafnhgUiN/r3trOi6kJ/d2jynzMVUc99GS2sicZB95z2uxYL434D2dNv30mTb+/bcvu8UN83FGvk7VX4dpuY88AlZsVOsVhcG4lU49mEjnXLexZAafQF00mDOzPp8eeyDR0W8dq6kqTCJd3BVXjF77c/Kz97wMJtIcG0yMm1k1HZ4/1DdhXav+Im65R1/u4rfbjy6YddRVFLBupITtPnLM76MDcaojuRO+WxsqCPiE05fVFvoYGipDo5bCQIKKkJ9VTZbb6VdbOwCr00tnDZGgz/UDO2IDsLrR2t872hmIp91Z1f2xFMOJNJms4VBPDGOs8tvFOuyRpPWdqsKBMR3W0/t7eLGtL6dTa/cEJvd2DjOYSLOioYLBRHpSQcu+WIrFNZZVs2eKomCM4UPf28xtv5tZCSzney6ri47rPnpoV6dVm2qGwjGUSLtuv2Ij64F4iqaqwtZeIR7ceSzHiuscSpDOmqKj4FTWEPQLoUDuLT09S2H0OxTzf8ftDh9GO3MgJzU7lsoQDfqpDPvHdQc5I+Qj9uc0VlkVAqaytjOMui67h5JF7wfnWIf7pzbwSKaz/Nuvdxa1YH701EE+t3H7uHMkXPdR3jndb5+zhZI1paKA5T4aSqZJZ43rpskactxHAG8+ayk3XLjSdRnl01gZdmMBxwbjLKoO01xluYac0fcjdszh1CU17vucDh8sYamvCOaMdoYSKZbURvD7hK6hJK/+4oN8+9F9HOyxLqae4WTRDtsShQCV4UBOh5XNGnYdtTJUdnSMZhZ5ReGFtj4Arjh1ETA5F1L/SJJTltRQFQ5M2VLoGU4ylEhPufBgPs6ovKUuOq776IVDfXQPJ9mRN2rsG0nyDz9/adJiMRRPs6TW+p2HE2l2dAy47qJ4KkM6k2UwnqapKkQk6Jvwc3cfG+KPv7OJXzx/2N3miEExKyqdyRLw+Qj68yyFaWTieIW0WFwhlsq45V2818z+fFEI+akIBcadvOYMpDrsjrrRFs/JTHhLprP85+9eZiSZdgtPJjPZogkGPXkCNFmePdjLbQ/s5uFdhdc0OTYYtzwM44jwaKA599j7bI/BdDLFSoGKAlZcwBlYOKM5ICfQDPDuDa3ccs3pRT/HcR8ZYzg2mGBRdcQVGbAmyW2x8/fPaPGKwqilUFsRpLEq7HYCqUyWeCpLdThAXTTI43u66BpK8vT+Hg548p2dDjyfkWSaipCf6jxRcKrAAuzoGO0UnXjCukWjCwu9+uRmfALffXwfzxwYf82AvliK+oogaxdVsXuKnbtz7EITAafCUCJN0C80V4fHddU4HcOeztzjPbSrk/958gA/ePLApI/nWEf9sRRvu/1xvvHwXgDeeNsj/OcDuxm0Y1S10eCEGVGFMr66BsfvzFIZy1LwioLfJ9NyHw3G0zTaHX6xDKSRZIaaSJCAT2jzCNVez28XT2aI2JbCeCmmjtvnsMd9BJObq7B5fw9f+s0uHtzRSZ/HtVosnuUVhakkQjjv6ypyPp3jjRecdwYr+ZbCvi7rPmmfodt0tlBRINcP21Q1WtyuOs9SmIiGyhCJtFUuo3MwQXN12DWF/T7hLa9Y5oqP1wW1vG5UFGoiQZqqQq6l4Ix0qyMBaqNBXmq3OortRwY40D3iltt4sa3wZDHHfeRYCs6NsMuz5rTXv9reFyMU8HH6shrXf72isYKPvf5kXmof4F1ffSLHAjjYPZJzI/SNpKiLBmmtj3Kkb2qjMWek1DWUnPQo/R9+/hIf+t7mnG2D8RRV4QA1kQCJdJZEunCH5LQ73zJxROL/PXlgQr+2MYahRNpNFtjTOUQslWFv1zAjyTR7O4d5/lAfg/G0+xtO9N0cH/vOowPuMbqHx7cUUpksQb+PoH+0MN1JzVXTDjQvr48SDviKu49sKyAa8ruWgk+sDu6HTx3kzk2HXHdsNBgoKgrGGHcZ3NGYgu0+mkQGkiOS7X0j9I2kWGQPwronEIVkOjslwXRFocj5cGJvxUTbGEN/LIXfJ/SNpNxr0hjD/i61FBYcUY87KMdSmIYogOXT7OiPs6gmTNDvo6EyxJkttZy3st7dd/3Swu6j2miQpqqwa2I6bojqSDAn8N3WG2Pr4X5WNVWyfmkNLxwqbCnEkpZftyoSIJMdXZJz19FBtx3bj+S6j1rqoiypHRWqxTURbrpiHQ98/NWICD/zpB7e9KNn+fQvXgKsZSkH4ilqK0JWfGWKpb69PtX9k7AW4qkMP322jQd2HM0pKz4UT+ecr2IupFFLIVcU9nUNI2KNXH+7ffySCbFUhqyBxbb7yDmv7X0xt7M80D1sz3sJUBOZWBQcv/dO263XH0u5KafFOp10xhDwCyLiCsNpS6vpHUlOuRbQQDxFTTTIoprwuO6jaNBPRcjvpr+esqSGF9v6+cy9W/nO4/sZSXpiCsnCa4QPJzPuJM8j07AUnGO398boH0m5Ze6LTWDzCsFUXEjOb9JdIMU3kx11G3UUOV8jyQyZrGGlndruWGBdQ0nXstWYwgIiXEwUolOrLO6Y3DuPDhJLZdwL4MOvXsNHLl/LyYurAaivCLLMtg78PmFxrfeYQU5eXM2BnhEGPZlA1ZEAdRXBnOM8c6CXlQ0VvGJ5LVva+92RvTdDZTiZpjIcoCpsfRfn83Z2DNJSF+X8VfXs8BQRa+uzRcF2h1R53ruoJsJl65q4+7l2d//23hgH7ZLiA7EUxljfr6EyRN9IakoZJDlB7q6JXU+/39nJSNLqlB/dPVprySlR4oh6MRdSMUthb+cQl57UREtdlJ9sbiv0VhcnqO1YCo4Fdrgv5rpVDvXG6Bu2OtrJWApOB9M1ZKVpeoPLR4oESFPZrOs6clKnT11qWXuTScsdTqS59iuP8vS+Hms2fzTI4uqIOwLOJ5a0LIXKUMC93s5dUceR/jiJdJbDfTFiKct9VBEKFK2m6k3DdL6nc31PJtDsWBcHekYYTKQ9olDEUvAMHqaS5ea8z3HjeekdSbrnoKPI7+PEadba7XMsMMdVet7KenpHUjlCaIzhh08dnPPqxioK5FkK1TO3FJ47aKWWrbAziW68bC1Xnr6EkxdbF8Si6ggNlSF8AjWRAOGA310BrjYa5OzWOjeraMBjKdTaI993bbBWOc0aWNlYwenLahlJZjjUM8KTe7s585b7XZeQc/M6Hbtz0e06OsgpS6o5bWmNu9IcWB3z8vqoGzhdVDN6PgDefu5yjvTHeXJvN6lMlp6RpHsjONkfdbYoeLfls2l/z5iRYFtvjLXNlZYbonOspRBLZvj2o/vc1OGNW45QXxGkJhLICQAOJixXjSPqheofxVMZeoatyYJtvTE3p94Yw76uYdY2V3Hy4iqO5gUFB+K5RQmdG7a+IkQo4HNv8mODCdfayWSN2yavKBzqGeFNtz0ypnPydpQ7OwbptDuiUMDnLv6UjxVTsK6hYMBHU1XIjXMUGt129MdzrLFnDvTyQls/j+3uYiCepiZiWwqDxSyFLJGg363KGvL7XJdo0C/0x1J0DyXsQLO1T6GRf28Ba7JxCpaCM9p3rvfVTdb1U0wUeoeTbobWVCyFnnEsBa9wdvQXPq4TR3JEy3mP8xu8ap219rw3rvDcoT7+7u4t3DVOynkpUFGAnGyi5pyYwtQsBcfKeN525axoqMx5vToSZHl9lEU1Yfw+oaEy7Hb0TvyiJhLkFa117ufkWAr2vm86c6n7vhWNFay0xedgzwjPH+pjJJnh8/ftAOyYQtCKKYDViaUyWfZ2DnPyYksUwFolLpbM0DWUoKUuyiJ75OuMgB1ev34xVeEAv9xyhJ7hJMZYs7zjqYzrwqmLhqivsM5j73CSTft7uHPT6IV9qGeEd331Cf7lVztyPrutN8bqpiqW11fkBCwd7t/awa2/3Mbf3b2FoUSa320/yhtOX8Kl65p4eFeX21k7/vvqcSwF56Y8f3UDxoyO2I4OJBhJZljbXElVJJjj137uYC/n3vobPnbnC66IOKLgWFTOiNEY2JyXd15ju7Sc9mw+0MPWwwM8aJdGcegeTrq/746OAbeDO21pTVFLwco+stxGQb+PxTURGiqt67GQ7/wz927lph896z53cuQP9ozYlkKAReNYCnGP+wigvjLI+avqaa4O86FXrXE/y7tPobiC0zZnjZ5wwOfed5MpddEx4Ey0tNrZUGm5LsdzH61pqiTolymlpfZ4rLd8nCBzwCdF3W3O4G6dLQqdttju6x4m6BcuWN0A5LpQnUSDZw8Wdg2XChUFcgPN3mwhrw9/Mjij4xcO9SECrQ3RMft8/u1n8fErT3GP5RyjJhog5PcRCfqojQZZ01zJcwd7XVGoiQRZt7iaZbURTltazWlLLVfUyoZKVjZa4nOgZ8TNSHpoVyePvNxpBfvCAaptUfju4/t5zzeeIpnJcmZLLScvrsYn1kjLCSCvXVTlisHiPFGIBP2sW1zFge7hnMyUowNx1yqo9VgKPcNJvvXIPv7u7i1unMQpKnjXM20MxFNkswZjDO19lpWyuqmSfV3DpDJZXmzr43fbj2KMcS2wu59r55LPP8BwMsPbzmnhsnXNdAzE3WypoYRVt8qx9Aq5T5zO9ZK1jcCoC8lxW61uqqIqHHDfa4zhnzfuIBTwcfdz7dz0w+esY9mvV0UCVIatzs/p4J7e1+Oedxi19gbteSbOSHXz/lzx6B1Jsra5kqaqEDs7Bl1ROKullmODiYKz0FMZQ8DjPlpSE3HdMN3DyTHB9iP98Ryr41n73L58bJBEOktNJEhjZYihRLpghVwnVuXUWqqvCLFucTWbPvU6N4U5lTE5+xQSBcdSWGbHsKKh0QGMt4heW+8IX/jVjjFpqh15o/26ihBNnuy9fHqGkzRWhVhSG5kwEcI707i3QF2loUSaYwNx9z44ZUl1UdF20mUdS8Z1H3UO09owOrDzBpud0jLPHZrbSW0lXY7zeMHrPqqNWml26ayZsqVQEbLcQMPJDMtqI4QDY+czXLquyf3/jWcscf+3AqMpd1nDs1vreHhXJ5ec1GS/HuCGC1dy/QUr8PuEU5fU8OTeHlY2VrCoOkwo4ONQzwgHuoc5fVkNvcNJ/v23L7vtWruoitaGKD97to3FNRH+8S3rufqMJfh8wqqmSrYfGXBTY09ZUk2z/ZktdWOFbVltlG1HBnLS/o70x90Lvy4adM9pz3CSIwNx0lnDXc+08ZHLT2LjliM0V4fpHExwyy+28vDLXbzzvOUMJdIsr48iAk/s7eaSzz/g3jx3/ulFPHeojwtWN7CoOkxHf5xPXHWq9dwWrucP9nHy4mqGnJiC6z4aayk4AcGL1jYiAnuOWZaCYzGsaa6kOhJwi909sOMYT+/v4bNvPYNth/vZuMWajJhrKQSBGKcsrmZHxyCdgwkuXNPAlrZ+hpMZqiMB4qlR68XplDbt72EgnuLLv9nFX752HT3DKVrqIlSEAuw8OuhalqctrcEYS4C9yQlgLccZsgPM7zi3hbWLqlxh/tVLHXz4B89QFw3ywUtXc9MV6+gZttZMSGWy+ER4zh6N7uqwRLEmGnQtr/5YKmewBKMz5R0rwIl3AW68DCAS8lNhi2WhkX+PnXm0prmS9r4Y0aB/zETAbNbwVz9+nk37e1laG+G9F60CrBn9XUPJnPTpumiQxqpQcffRSIpldVFSGVO0A3fYtK+X324/yqb9PW6G1GA8TdyOlXzq7i08e7CX6y9YAcCZLbX88sXCJWeca7C+IkRjVXjUfdQ9zJqmSpqrwoQDvhxRcOYFHeqJudmMc4FaCuS6j6L2RK9wwFewUx8PEXFHZ048YTw++tp1fPS16wArtuC1TM5praNrKMkOOzPIWezHWd/hbee08K7zltNSF8XnE1rroxzstiyFkxdXc/WZS12XQEXIz+KaCI984gp2f+6NPH7zFfzxJavx+ZwslRq2dwywo2OQcMDHqsZKgn4fd9x4IR961eox7V5WF+FwX4zOHF9qfNR9ZGcfATkxhx9vOsTB7hFebOvng5eu5twVdfzsuXZ6R5J87eE9gDVn49Ql1STTWZbVRfn3PzibkN/HPS+0s+3wAOetrOcrf3gud/3Zxa7J3VIXxSdWkNwY47qP6qJW3GbT/h4G4ylu+uGz/PN9261Cf/YIc3VTJS11Ubc0+N7OYSJBa6RdFQ4QT2VJZbLcufkQS2sjXHd+K0tro/THUjklzi1RsK6XDatGs8yW11e4lpwTaAaro3U6pbbeGP+8cQffeWw/v9/ZSc9wwspYW17LtsMDHOyJ0VgZYlmdJX5eX/gPnjzANV95lJFkxrUUPnblKVx7dov7G9z9XDv1FSHqKkL8ry1mjjukZzjJrqODDCXSrG2udDOBauz4h9PWfOJ5MQXvOuWLq8PubOeKoN+tx+QtihdLZjjcF6N3OInfJ27ByWjQj98nRIN+t43ffHQvm/b3UhsN8t3H97tJDk7H6s3qq6sIjmspdA8lrHNZGykYn+kcTPC5/91GMp11LduDPSN0DyfcQWLPcJJYMsOvtx7lUE+M5w72URUOsLqp0l6LxTpf3iVN3ZUco5YL+UDPMNmsYX/3MKsaKxERWuqibrJFOpNlx5EB15XsWMlzgYoCue6jaNBvjzKn5jpyaLBjEivz4gkT8c7zlnPDK1e6z51KrL94oZ1o0D9mpuorWuv44rte4XbsKxoqePnYIIf7Y6xoqODyU5rdfR3zHcDnkzGLrK9fWsOhnhibD/Ry8uJqV3jOXVFPXUWIfJbWRkmks276JVgdVe+Is4TpaKZU52CCzsEEqxorONA9wg3fegqw4iKffsvpfOjS1fz0zy7GadHy+grefu5yfv7nl3D3Ry7mree0cPFJjdy5qY101nB269i6U6GA5UNv741ZNXOyhqpIgGjIz42XreVnz7bzptse5b6XOvjGw3t5/b89xNbDA1SG/FRHgpyyuNr9Lvu6hlndVIXPJznB+Z7hpCuWztyT3pHkqChEAq7bY1VjpZsv31JnucPAsvaW2gH8tt4Yh/vi7n4/evogYE3+6h1OUV8Z4uK1jaSzhgd3HKOpKuyOwJ3A9MYtR/iHX7zEi2397Do66MYUHCJBP5V2p33DhSu5cE0DnYMJEunRQnedgwk39vG2c1rc99Z4Unr7Yyn2dg7x5d/sIps1pDNZkpmsNf/Fvra810nAjmkARQPNH//J87zptkfoHExY2Wr2+50B2rkr6/jBUwd4538/zj9t3MHr1y/mM9eczt7OYf76rhd423895rq8ckXBdh8VyBJKZbIMxNPUV4ZYWhfl6EB8jBV55+ZDfOORfWxp73et1J0dg8RTWTd7sGsowUO7jrn1nx7e1UlzddhNzjg6EKdrKMEln3+Qbz1q1UMa8MQG1zZXsbdzmI6BOPFUllX29dFSH3VjCvu7h0mks7x7w3KCfpnTuIKKArmWQoV9EddM0XXk0GgH9yZjKXi59uwWPnDp6Kj89GU1fOTytVy8ton3vHLFhO9f2VjJns5hjIFVTRWcv6rBNcOdm7IYTnzihUN9nLKkesJjOZ3Ti+39VIUDVEcCHB2I0x+zcvEDfsvKqgoH2HV0kKyB9160iqvPWEJLXZQ/ffUaWhsqOLu1jr9/83rObq3jree0IGJZCkG/j7Nb61zxunL9EncEe04BUQDrfW29o8ubOkHmv77yZC5e20hb7wi3XXcO99x0KYOJNPe+eNi9iU9ZUs3ezmGSaWt05mSIONaZsyqe445yfuOuoYQbc/Cm7rbURd16Vi31UddfXBMJcpKdgbb72CBH+mO85pRF7u8UCvjY2t5PMpOlsTLEhpUNBP1WWfWmaksUROCFQ/0c6Y/xsTufd9f56BtJjRk4gDVICfqFGy5cQXNVmJ7hRE4sqGsowQuH+miqCrmuSrBiXI6lMBBL8csXj/Afv3uZ7R0DxO0YgzeI3JA3eHCukUjQ7yZg7LeXB33mQC8bt3TQO5LioV2d1FWE3EGEY3l8873n8+7zWtnS3s/HXn8yX/nDc3jjmUtZVB3mZ8+289zBPv7zAWt1wLOW1xHwCT6xym43VYWJpTKMJNO098V45OVO9xyBZdVcelITBnjTbY/wyZ9t4ZM/e5FEOsNDdhbbkf6Ya4k41QKc7MHuoST3vdRBfUWQqrA1QbK5OuzG4Y70x/n8fTvoGkqwaX+Pew4rQtbgbk1zJccGE+7qhGtsUVhWG3VndTuuo3NX1LN+aY0rgHOBxhQYFQURK/uhMhwgbzA9aRz30copikI+Pp/wiatOnfT+jvkNVtZTJOjnorWNPLDjmHujFeM0z0S6UyclCtbF/1J7P4trIgR8wpF+yx9c7+kcGipDbgbFysaKHNHL5zPXnM47zl1e0DJ53fpFfOrn1k2zKC/w7dBSF2XzgV722qmsTmcZ8Pv49vvPp6035nb2V52+hPte6sgRhXTW8PieLg73x/mQLTxOkHg4mWYglnY7SWfGbfdQkuFEmoBPCAd8o6JQH6WlLspzB/tYXh+luSpMdeQAi2vC7kzrLe0D9I6kaG2I8qp1TQzZ62o8Y9/89RUhoiE/57TW8/T+HpqrrPe+7ewWfvDUAfZ3D5PNwrfffz5XfvlhMnZBvHxeta6ZqnDALbmSNbnVazsHExzoHmZNU5Xr5gJLwBwrtD+Wcl05T+zpZlG1dd4ioVH3kTemAJYoPHOgl2jQz5LaCGe01LBxSwd/8qo1fOG+HTTakxs7BuJc0NDgnltHIKMhP19451l87m1nuG4xgB/+yYWks1luuWcrT+7tsY8VYWldhKF4Gp9PXEuuazDJNx7Zyw+eOsD/fOCVrk++odISwDv/9CI+cdcL3PfSEfpGUpy6pIZnbavpSF/cdR858YqTFln3RntfjN9tP8abz1pqi04XzdVhltrB8m88so+Hd3US9EvOBETnO65psq7DB3ZYWWeOpbCsLupactuODBDy+1jbXMWl65r479/v4UD3cM5vVCpKaimIyFUislNEdovIzQVe/5iIbBORF0XkdyKystDnlBpHFKJBPyLC9Re0cv35E4/OC+H4VqfqPpopKzyisMoWpFefbLmQKooU8HNYUhNxL9ipWAojyQxNdiZHh+0+8nYO9ZUhdz1apwMuRnUkmDNS9bKoOsLrT1vMlacvLvr+lvooR/rjbr76KYtHv0ck6HcFAeAjl58EjGZWOcUJ73jaSpt1SqO7aby2peCcIyePvns4QZ890UtE3P2X1UXdAP3yugpec+oiXvzHK6mOWPudtKiKR3dbI9KltVH+6z3n8t0/voDVTZXuaNbp2C6ys6Oaqq3nH3/DKQhWh/K+i1eytrnK/b0DBSyFf3rbmfzdG08DRjPrdnoKIHYNJTnUE6O1ocId+cLY+IcjCo/t7nLTcaOeeEF9vqVQO+o+AnjTmct4/lAfX3t4L0/v7+FjV57slpWvrwy6g4H8YpP53+mkRVWcuqSGt55tubosSzXIstqo+xnN9u/TOZTgcJ9VSfiv7nzejRs5Vs15K+v53ccv57l/eD0nLari8/ftcCsXH+6PuRaVMy3FsRTueqaNoUSaq89c6rqumqvCLK4NU1cR5OFdnZzdWscHLl3NgZ4RYslMzvosa+01U3634xjhgM+1MJbag62j/Ql2HBlk7aIqQgEf77toFQG/j6/Z9bRKTclEQUT8wO3A1cB64HoRWZ+323PABmPMWcBdwL+Uqj3jEbEnszijlD84fwXvPr91Wp+1rC5K0C+sbJqZpTBVHMukOhxwhelt57bwkcvXusGqYoiI60KajCg0VobcCUDWCCnCgZ4RnjnQy7pFo+9v8AjE0tqxWUxT4evv3cA/vqV4McLl9RVksoZHXu6kJhJwJ24V4szltfz9m07jPXYMZ02zlbf+2+1HCfl9rLc7qypvYDGVcW/qRo+l4JQFAavI4fqlNTRWhnj1yc1celKTa1V54zjrFlW7efVL6yIE/D5CAZ8be4DRTvZiWxScjq6lLspfvHYdLXVR/vw1lrg5vu5CloKXQqJwuC/G0cE4KxoqEBkN+NZEckXBScl8el+P64ePFgk0w+jAwbmn3nTmUgA+f98OTl9Ww3Xnr3AHAU5lYBi7HkQxrj5jqZV6a4vPBy5dzY2XWfMjHHdV11CCjoE4a5sr6Y+luOWerYA1WPEiIrz/4lVuraYVDRUc6bMWmPKWIm+pi1IR8vO8425b28iGlVayw6KaMOGAnyc/+Vp2/t+r+PmfX8I59iTUl48NMhAbXclxRWMFPrGstNVNla5F5lxH7X0xewJlpf3ZEd553nLu2tw2J+s7l9JSuADYbYzZa4xJAncA13p3MMY8aIxxZms8CSwvYXuKErALiRUriT0Vrr9gBffcdOmUZ0PPlFY7RXFFY4XbAdVEgnziqlMn9b0uWtPkpsZNhIi4I8HmqjBLaqP0jaQYSqR5/8Wr3P2cyVOhgM+96UuFc0M9vqebU5ZUjwmm5/OhV61xR3lB20xPZw1ntNS4WWeO+8gJ7Nba36E6bM0p6RpKcqh3xE3lfds5y9n4l69CRLj4pCZ+8KFXFhy9e62WZR6xXOURBSduce7Kev74klW87rRRK+nPX3MSD3/iNe7IeJ0tCgHf+LezU8bdKZVeHQnw/KE+jBmdU7OiIWrfC1YZ7oqQ356dbM0EHk5mXLdNNOQrmJIKHlGwX1/RWOGuWPjZt56B3ydcstYShXpvTGGS92BtRZAbLlzpzol4w+lL3NRQx6rqHkpydCDOBasb+MvXrnMnnjVWjnVRvv3cFmqjQS5e28TKxgoO9IzQO5LKWWWxoTLkCs6bz1pGwO/jvJX1XLSmkQvXWOIdCfrd68cR650dgzmWQjjgd8V3lccd5CQhHOgepq13xI01APzpZWtIZ7P8/Ln2SZ2fmVDKmEIL4J2f3T0bz7AAABFISURBVAa8cpz9PwjcV+gFEbkRuBFgxYrpuXUmIhLwTxiQnQzRkD/HRz9XREN+ltREckabU+GjV5zEn12+dsLO1GFpbZT93SM0VYVdd8orVzdwZs5NZN0ES2oik/7c6eJ0zIn0aJbIVDhliTW34JwVo5ksjqXglB5wRs4i4ubCt/fGeK3dMU0WJ9gMuW41bydQb5+7oN9X0ELyezKNTnEthfFFweksd3cO4RNY21zlBjsd9+Or1jUzksy4v5dTlqN3JMmrT27mt9uPcr+9YFQk6Of8VdW84fTFYyzMM1tqaW2IujN4Af7+zevZ2znEufY5Pn9VAysbKzijpZbaqNW2ieJfXj79lnzHg4VjtXT0x+gaSrK4JsKNl63h3hcOs/PoYMG4VUUowF0fvojaaJB//fVOHt/TDcB5KxvYtL/XLkljzYE42DPCNWcvc9v7oxsvLNiOlY2VhAM+dh0dpD+Wyrku1zRVcqB7JGcg4FjTT+7tJmtwV2F0Puvej16aU0izVCyIQLOI3ABsAF5d6HVjzNeBrwNs2LChJKvBe4Nmxyv/fcO5OQX9poLPJ4R8k++4nZFgc3XYzbT6E7u8gYNjpueXyigF3glTk3GB5ePcsN6lVqtcS8Ey2b3WX2NViB0dAyTS2TETySbiJPtmb6wM5VhxLbbr0XvsybXd+ryJ3EcVIStDaihhrZewqDrs+tCdkesNF67khgtHQ3uOKHTb5SEuWdvkFh+MBq0R79f+aMOYYy2ptebFeDl/VQPnr2pwn0dDfh76m9cAo2tMT9ZSGI9wwMoedGYEL6mJEPT7+OoN5/Hswd4xq9M5OBbX0tqoW67k7NY6fGJZMz6fsKKhgv5YqmgWnBe/T1i3uIr7XurgcF8sJxlkTXMVD+7szBkIREN+GipDPLrbEqT8AV6xFR9nm1KKQjvgdcwvt7flICKvAz4FvNoYMz8rvWP5MmfjgpxPvKPcUuP4ypurw1y0ppH7/89lYzpjJ6A3UZB5NogE/e4saW9cY7K85pRF3PvCYS6y3QCAG0R1qp165640VoZ5zO4cC5UzGY+WuqibleMl4PfR2lDBcCI9JctqlV3LZyJLAazfayiRtlwhdowhHPAVdRvWRIIc6Y+RTGdpqAzxkcvXjorCLA6ign4fH73iJNcdNFOaqsPu2iNOWfNVTZU5I/NiONc2WL/VMvv3Arj12jNIprOT/n1OXlzNz55tZ0VDhRvzACuO5bTJy9LaiJuOurp5bpNVHEopCpuAdSKyGksMrgP+0LuDiJwDfA24yhhzbOxHzB3eoJkyMc7IvKkqjIgUHJ07ZvxciAJYN3DnYMIdOU+F9ctq+NX/uSxnmzOB7XCe+wgsS8EZZU/VUvD5hHNW1OVYNw5nLKsdU89nIoJ+H59/+1mTcls2V4XZ1zVMQ2XIFYLWhgo32JlPTTTojrjrK0NctLaRs1vreP5QHxXB2e0+nJpgs0FTVZinO63Yx+LqqV1/3t9lUU2Ys5bXkrXLP9VOcVLr2a113PP8Yf79urNzrL+rz1hKW29szGTMZXVRth4eoKkqPOdxSYeSiYIxJi0iNwH3A37g28aYrSJyK7DZGHMP8EWgCviJrbwHjTHXlKpN4/HBS1e7fk1lYl532mJ2Xjw4bkfUMIfuI7BScY/0x9wYx2xQFQ64dZK862t43XSF6kNNxLfff37BuTD/9PYzyWSm7iF9x3mTy9FwMpAaq0Ythdb64u2vjQbd2c+NlSFEhJuvPpUv/XrnmLLqCwnvCopTHZQ4vn0R6zt/+Q/OnnY7/vCCFbzh9CVjCks2VIb42wLzkJwEjjXTjA3OBiWNKRhjNgIb87Z92vP/60p5/KnwB9Ocl1CuNFeHx12vGqyMk0jQxxktc+ML/durT53WmsTj4VQ+hTxLwRa8hsqQOz9hKhTLCJtKLGE6eCdwOWXivXNc8vF+ZydGdOGaRn7y4YtL2MqZ44j2dDLfHPdRU1WYgN83o07SW/Jjcse2BGnNPLmOYIEEmpUTk0XVEbZ+5qqcTJlSsrQ2OuP5EPlURUbz570FEh1rZPk4o+yFiCsKFSH3/9ZJikKhVM6FiiMKi2vCU858qwhZJT4WzVFVUi9LbVGYbhbhbKCioJSUuRKEUuHMVcj37zoT2FqnGE+Yb7yWwtrmKlY3VfLK1Y1F96/1uMzyJ30tZBxRmK7rcnVTpTtvYC5x3EbTyaCbLVQUFGUcHHdOfoCxqfI4txSqwtRVhHjwry8fd39nwl7QLzkLBi10HNGeiuvGy1dvOI/ABCm+peCMllruuekSzpwjl2shjp9fWVHmAWcCW74oLKmN4JP59f1Oh9OX1nDSoirOmmSn43zv+opQyScgziYztRTmKmOuEGctn3gORClRUVCUcfAWiPPSXB3m3o9eOq3Z0/PJopoIv/1YwTmiBXFEIb+20ULHiQfMZ+d+vKKioCjjUF3EUoC5m2E6nxyvorC8Pso/vHk917xi2Xw35bhDRUFRxqFYTKFcqDlORUFE+OA463coxdGV1xRlHCrd7KPyHD8dr5aCMn1UFBRlHBz30XTX7D7eCQf8vO60RVy8tvACSMqJR3kOfxRlkpS7+wjgm+87f76boMwhaikoyjgUyz5SlBMVFQVFGYdXtNZx42Vr3GUxFeVER91HijIOkaDfXfheUcoBtRQURVEUFxUFRVEUxUVFQVEURXFRUVAURVFcVBQURVEUFxUFRVEUxUVFQVEURXFRUVAURVFcxBgz322YEiLSCRyY5tubgK5ZbM5sslDbpu2aGtquqbNQ23aitWulMaZ5op2OO1GYCSKy2RizYb7bUYiF2jZt19TQdk2dhdq2cm2Xuo8URVEUFxUFRVEUxaXcROHr892AcViobdN2TQ1t19RZqG0ry3aVVUxBURRFGZ9ysxQURVGUcVBRUBRFUVzKRhRE5CoR2Skiu0Xk5nlsR6uIPCgi20Rkq4j8pb39FhFpF5Hn7ccb56Ft+0Vki338zfa2BhH5jYi8bP+tn+M2neI5J8+LyICI/J/5Ol8i8m0ROSYiL3m2FTxHYnGbfc29KCLnznG7vigiO+xj3y0idfb2VSIS85y7r85xu4r+diLySft87RSRN5SqXeO07ceedu0Xkeft7XNyzsbpH+buGjPGnPAPwA/sAdYAIeAFYP08tWUpcK79fzWwC1gP3AL89Tyfp/1AU962fwFutv+/GfjCPP+OHcDK+TpfwGXAucBLE50j4I3AfYAAFwJPzXG7rgQC9v9f8LRrlXe/eThfBX87+z54AQgDq+171j+Xbct7/UvAp+fynI3TP8zZNVYulsIFwG5jzF5jTBK4A7h2PhpijDlijHnW/n8Q2A60zEdbJsm1wPfs/78HvHUe2/JaYI8xZroz2meMMeZhoCdvc7FzdC3wfWPxJFAnIkvnql3GmF8bY9L20yeB5aU49lTbNQ7XAncYYxLGmH3Abqx7d87bJiICvBv4UamOX6RNxfqHObvGykUUWoBDnudtLICOWERWAecAT9mbbrJNwG/PtZvGxgC/FpFnRORGe9tiY8wR+/8OYPE8tMvhOnJv0vk+Xw7FztFCuu4+8P/bO9dYK6orjv/+aiuWIklBDU1tQJQS20TwkZQChlrSSKPEIkZbIxiNrwim8oEYSU0/1Ib4aEJSW1KquQmlDb6oV9uowRcGJTyuvBQRo8YoeLFqVAoq4PLD2ucwjDOHS7hn5hbWL7k5c9bZs2fttefutfc6Z9bGZ5QNhkl6SdJzksbXoE9R3/Ule40Hus1sS0ZWqc1y40Nl99iR4hT6HJK+DTwE/MbMPgH+AgwHRgHb8KVr1YwzszOBScCNks7Nfmi+Xq3lN8ySvglMBh5Ior5gr69Rp43KkDQH2AMsSqJtwPfNbDQwC/iHpOMrVKlP9l2OX7H/BKRSmxWMD03afY8dKU7hXeDkzPvvJVktSPoG3uGLzOxhADPrNrO9ZvYlsIA2LpvLMLN30+t2YEnSobuxHE2v26vWKzEJ6DKz7qRj7fbKUGaj2u87SVcCFwCXp8GEFJ75IB2vwWP3I6rSqUXf1W4vAEnHAFOAxQ1ZlTYrGh+o8B47UpzCKuA0ScPSjPMyoLMORVKs8l5gk5n9MSPPxgF/CWzMn9tmvfpLGtA4xr+k3IjbaXoqNh14pEq9Muw3c6vbXjnKbNQJTEu/EPkx8HEmBNB2JJ0PzAYmm9nOjPwESUen41OA04A3KtSrrO86gcskHStpWNJrZVV6ZZgIvGpm7zQEVdmsbHygynus3d+m95U//Fv613APP6dGPcbhS7/1wNr09wtgIbAhyTuBIRXrdQr+y491wMsNGwGDgKeALcBS4Ds12Kw/8AEwMCOrxV64Y9oG7Mbjt1eX2Qj/Rcg96Z7bAJxdsV6v4/Hmxn02P5W9OPXxWqALuLBivUr7DpiT7LUZmFR1XyZ5B3B9rmwlNmsxPlR2j0WaiyAIgqDJkRI+CoIgCHpAOIUgCIKgSTiFIAiCoEk4hSAIgqBJOIUgCIKgSTiFoC1IeiG9DpX0616u+9aia7ULSRdJuq1Nde9oU70TJD12iHV0SJra4vMZkq46lGsEfY9wCkFbMLOfpMOhwEE5hfREaSv2cwqZa7WL2cCfD7WSHrSr7fSyDvcBM3uxvqAPEE4haAuZGfBcYHzKQX+zpKPlef5XpYRo16XyEyQ9L6kTeCXJ/pWS873cSNAnaS5wXKpvUfZa6anOOyVtlO8LcWmm7mclPSjfX2BRenIUSXPluevXS7qroB0jgM/N7L/pfYek+ZJWS3pN0gVJ3uN2FVzjdknrJK2QdFLmOlMzZXZk6itry/lJ1oWnaWic+ztJCyUtBxa20FWS/iTfy2ApcGKmjq/Zyfwp6bck1ZliJOhlap+5BIc9t+C58xuD57X4o/jnSDoWWC7pyVT2TOBH5mmTAa4ysw8lHQeskvSQmd0iaYaZjSq41hQ8ydoZwOB0zrL02Wjgh8BWYDkwVtImPM3CSDMzpU1ocozFn2DNMhTP1zMceEbSqcC0g2hXlv7ACjObI+kO4Brg9wXlshS1ZTWeR+g8/EnmxblzTscTHu5q0QejgR+ksifhTuw+SYNa2Gk1nlG0jnQUQRuIlUJQNT/Hc7WsxVMCD8LzyACszA2cN0lah+8FcHKmXBnjgH+aJ1vrBp4DzsnU/Y55Era1+MD+MfAZcK+kKcDOgjqHAO/nZPeb2ZfmaZXfAEYeZLuyfAE0Yv9rkl4HoqgtI4E3zWyLeZqCv+fO6TSzXem4TNdz2We/rcDTqXwrO20HvtsDnYP/E2KlEFSNgJlm9sR+QmkC8L/c+4nAGDPbKelZoN8hXPfzzPFefEeyPSn08TNgKjADn2ln2QUMzMnyuWGMHrargN22L9fMXvb9T+4hTdokHYXvGFjalhb1N8jqUKZr4ZamB7BTP9xGwWFCrBSCdvMpvq1ggyeAG+TpgZE0Qp6VNc9A4KPkEEbiWw022N04P8fzwKUpZn4CPvMtDWvIc9YPNLP/ADfjYac8m4BTc7JLJB0laTieSHDzQbSrp7wFnJWOJwNF7c3yKjA06QSeVbaMMl2Xsc9+Q4Cfps9b2WkE9WaoDXqZWCkE7WY9sDeFgTqAeXi4oyt9Qfo+xVt8Pg5cn+L+m/EQUoO/AusldZnZ5Rn5EmAMnunVgNlm9l5yKkUMAB6R1A+fPc8qKLMMuFuSMjP6t3FnczyeTfMzSX/rYbt6yoKk2zrcFq1WGyQdrgX+LWkn7iAHlBQv03UJvgJ4JbXxxVS+lZ3G4nsuB4cJkSU1CA6ApHnAo2a2VFIH8JiZPVizWrUjaTQwy8yuqFuXoPeI8FEQHJg/AN+qW4k+yGDgt3UrEfQusVIIgiAImsRKIQiCIGgSTiEIgiBoEk4hCIIgaBJOIQiCIGgSTiEIgiBo8hU6xWm6d3MzEQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f82f0054390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "costs = np.squeeze(costs)\n",
    "plt.plot(costs)\n",
    "plt.ylabel('cost')\n",
    "plt.xlabel('iterations (per hundreds)')\n",
    "plt.title(\"Learning rate = 0.0075\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "读者可以看到图中成本在刚开始收敛较快，随着迭代次数变多，收敛速度变慢，最终收敛到一个较小值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
